Notes on Boltzmann Machines

I. INTRODUCTION Boltzmann machines are probability distributions on high dimensional binary vectors which are analogous to Gaussian Markov Random Fields in that they are fully determined by first and second order moments. A key difference however is that augmenting Boltzmann machines with hidden variables enlarges the class of distributions that can be modeled, so that in principle it is possible to model distributions of arbitrary complexity [1]. (On the other hand, marginalizing over hidden variables in a Gaussian distribution merely gives another Gaussian.) Training Boltzmann machines still seems to be more of an art than a science, but a variational Bayes expectation maximization algorithm has been developed which deals with this problem in a reasonably efficient way for a class of sparsely connected Boltzmann machines that includes the deep Boltzmann machines studied in [2]. I will explain how this approach provides a principled framework for constructing complex Boltzmann machines incrementally and how it can be extended straightforwardly to handle higher order Boltzmann machines (e.g. Boltzmann machines defined by third order moments). The binary/Gaussian distinction is not an exclusive dichotomy: hybrid models containing both types of variable can be constructed. The term Gaussian-Bernoulli Boltzmann machine is usually used to refer to a model in which the vector of visible variables v is continuous and the vector of hidden variables h is binary. For each binary configuration h, the conditional distribution of v given h is Gaussian so writing